Papers with intermediate pre-training
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end questionanswering assume that pre-trained language can decompose complex tasks into more straightforward sub-tasks. |
| Approach: | They propose to use distant supervision to train decomposition-based transformers for large-scale parallel news. |
| Outcome: | The proposed model improves on semantic parsing and on hotpotQA and strategyQA datasets by 20% to 30%. |
Silver Syntax Pre-training for Cross-Domain Relation Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Relation Extraction (RE) is the task of extracting structured knowledge from unstructured text. |
| Approach: | They exploit the affinity between syntactic structure and semantic RE to obtain low-cost pre-training data. |
| Outcome: | The proposed model outperforms baseline models in five out of six cross-domain setups without additional annotated data. |
Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering (2022.acl-long)
Copied to clipboard
Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu, Xin Jiang, Qun Liu, Lei Chen
| Challenge: | Existing methods to train dense passage retrieval have a large data gap between upstream and downstream relevance. |
| Approach: | They propose a method to pre-train the dense retriever with the text relevance induced by hyperlinks within Web documents. |
| Outcome: | The proposed method outperforms existing methods under different scenarios and in the open-domain question answering domain. |
Learning Relational Decomposition of Queries for Question Answering from Tables (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Table Question-Answering focus on generating answers directly from inputs, but there are limitations when executing numerical operations. |
| Approach: | They propose to imitate a restricted subset of SQL-like algebraic operations and use them to generate a query. |
| Outcome: | The proposed methods bridge the gap between semantic parsing and direct answering methods and offer valuable insights into which types of operations should be predicted by a generative architecture and which should be executed by an external algorithm. |
On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)
Copied to clipboard
Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa
| Challenge: | Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models. |
| Approach: | They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. |
| Outcome: | The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task. |
Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Inductive biases play a critical role in NLP, especially in learning from limited data and generalizing systematically outside of the training distribution. |
| Approach: | They propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform syntactic transformations of dependency trees given a description of the transformation. |
| Outcome: | The proposed model can perform syntactic transformations and generalize semantic parsing with attention heads that keep track of which syntaktic transformation needs to be applied to which token. |